Changed SystemInfo dataclass to be mutable and replaced usage of _replace with dataclasses.replace. Simplified config validation logic in ConfigBuilder by removing the nested key check and directly checking for required keys.

Enhanced error handling in DockerDatabaseStrategy by adding user-friendly messages when Docker is not installed or the daemon is not running. Instructions for installing and starting Docker are now provided to assist users in resolving setup issues.

Enhanced the _add_fallback_spyglass_config method to add fallback directory configurations for kachery, DeepLabCut (dlc), and Moseq, in addition to Spyglass. Also updated the installer output to clarify that SpyglassConfig warnings are normal during setup.

Replaces the custom ConfigBuilder with a new SpyglassConfigManager that leverages the official SpyglassConfig class for configuration creation and saving. Updates the quickstart workflow to use this new approach, simplifies config validation, and improves validation script execution by running it directly in the current Python process. This change ensures better alignment with official Spyglass practices and reduces maintenance of custom config logic.

Add a check to ensure that the database setup strategy is not None before calling its setup method. Also update the return type of _select_database_strategy to Optional[DatabaseSetupStrategy].

Introduces an interactive prompt for users to select the installation type (minimal, full, or pipeline-specific) if not specified via command line arguments. Updates help messages to clarify that users will be prompted if no installation type is provided, and adds detailed selection logic for pipeline-specific installations.

Introduces a user prompt to confirm or customize the environment name during pipeline-specific and standard installations. This improves clarity and flexibility for users creating new environments.

Refactors quickstart to call install_additional_deps() only when the conda environment is created or updated, avoiding unnecessary installations when keeping an existing environment. The create_environment method now returns a boolean indicating if changes were made.

Removed unused Protocols and ABC imports, simplifying helper functions for command execution and file system operations. Introduced a custom exception hierarchy for clearer error handling and replaced generic RuntimeError with specific exceptions. Refactored setup step execution for better readability and added timeout handling to environment creation. Simplified validation script execution and updated directory creation logic to use new helpers.

Replaced uses of installer.command_runner.run with run_command for Docker operations to standardize command execution. Increased the default timeout for environment creation commands from 10 to 30 minutes. Updated validation and integration test scripts to run within the spyglass conda environment, improving reliability and consistency.

Added a validate_base_dir function to ensure the base directory is valid and secure. Refactored message printing to use a common formatting method. Enhanced error messages for missing files and validation scripts, and improved pipeline environment selection logic. Updated main() to validate the base directory and handle errors gracefully.

Replaces database setup strategy classes with simple functions and a mapping dictionary, and adds user-selectable configuration file location. Refactors config creation to allow specifying the directory for dj_local_conf.json, and updates user prompts and constants for clarity and maintainability.

Deleted the scripts/quickstart.sh file, which previously provided a one-command setup for Spyglass installation and environment configuration.

Refactored the quickstart script to introduce a modular architecture with UserInterface, EnvironmentManager, SystemDetector, and QuickstartOrchestrator classes. This improves separation of concerns, testability, and maintainability. Database setup and environment creation logic were updated to use the new orchestrator and UI classes, and interactive flows were streamlined. Legacy monolithic logic in SpyglassQuickstart was removed or migrated to the new structure.

Removed an unnecessary comment about ABC import in quickstart.py and added a filter to suppress pkg_resources deprecation warnings in validate_spyglass.py.

Enhances the quickstart script with better MySQL readiness checks for Docker, improved error diagnostics during environment creation, and more robust command execution within conda environments. Also refines configuration validation, output messaging, and system info wiring between orchestrator and environment manager.

Adds a database connection test before saving credentials, with clearer user prompts and input validation for host, port, username, and password. Enhances environment validation by attempting to locate the Python executable directly in the conda environment before falling back to 'conda run', and improves error reporting for validation failures.

Updated the quickstart script to filter out conda's misleading error messages from stderr output, reducing noise and improving clarity for users. Legitimate stderr content such as deprecation warnings will still be displayed.

This update ensures all relevant directories are created with parents as needed, resolves and expands base directory paths, and adds support for reading the kachery_zone from environment variables or config. It also improves error logging and masks database passwords in config warnings.

Introduces environment-min.yml for core dependencies only. Updates quickstart.py to select the minimal environment file for minimal installs and clarifies messages for full and minimal environment selections. Adjusts full dependency installation to match new environment logic.

The _find_config_file method now checks additional locations for the DataJoint config file, including an environment variable override, repo root, and the SpyglassConfig base directory. This improves flexibility and robustness in locating configuration files.

The connection success message now includes both host and port, as well as the username used for the connection. This provides clearer context for debugging and validation.

Updated the kachery_zone configuration to use only the KACHERY_ZONE environment variable with a default fallback, removing the DataJoint config lookup.

Introduces --yes/-y flag for non-interactive mode and --db-port for specifying the MySQL host port. Updates SetupConfig, UserInterface, and argument parsing to support these options, enabling easier automation and configuration.

Introduces a --config-file argument to allow users to explicitly specify the DataJoint config file location. Updates config file search logic to prioritize the specified file, warn about multiple config files, and improve messaging when no config is found.

Introduces an init_spyglass_settings() function for explicit initialization and adds SPYGLASS_AUTO_INIT environment variable to control auto-initialization at import time. Also logs when a supplied base_dir causes environment variable overrides to be ignored, and improves path resolution in output file handling.

Included the repository root directory as a fallback location for 'dj_local_conf.json' to support quickstart-generated configs. This enhances configuration file discovery for the validator script.

Adds a check to ensure the database port is not already in use before starting a new Docker container. Enhances error reporting by preserving original exception context during environment creation. Implements auto-accept for environment update prompts when --yes is specified, and refines installation type detection logic.

Introduces config_schema.yml as the single source of truth for Spyglass directory structure and TLS configuration. This file defines environment variable mappings for core directories and specifies TLS settings for secure database connections.

Replaces config_schema.yml with config_schema.json as the single source of truth for Spyglass directory structure. Updates scripts/install.py and src/spyglass/settings.py to load and validate the directory schema from JSON, ensuring DRY architecture and consistency. Adds comprehensive tests to verify schema validity, installer/settings consistency, and backwards compatibility. Updates .gitignore to allow config_schema.json.

Introduces _TestDatabaseManager to manage test database connections, providing a minimal interface compatible with the previous DockerMySQLManager. This change improves compatibility with GitHub Actions service containers and simplifies test setup by removing Docker-specific logic.

The _TestDatabaseManager was receiving port=None from pytest but trying to convert it to int, causing TypeError in CI tests. Changed __init__ to accept port=None and default to 3308 (GitHub Actions service container port).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

Tests were failing with AttributeError: '_TestDatabaseManager' object has no attribute 'connected'. Added the connected property that returns True for service containers (always ready) and False for null_server.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

The connected property was returning False when --no-docker flag was used (as in CI), even though the MySQL service container was available. Changed to actually verify the DataJoint connection works, matching the behavior of the original DockerMySQLManager.

Fixes test failures: ConnectionError: No server connection.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

test_db_settings.py accesses docker_server.container.id to check if running in --no-docker mode. Added container property that returns None for service containers, maintaining compatibility with existing tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

Changed wait() from a no-op pass statement to actually polling for database connectivity with retries. This is more robust than assuming the service container is immediately ready when health checks pass - there can be race conditions between Docker health checks and MySQL's actual readiness to accept connections.

The old DockerMySQLManager also returned immediately for null_server mode, but this implementation is safer by actually verifying connectivity.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

The wait() and connected() methods were calling dj.config.update() and
dj.conn() before test fixtures properly initialized the config. This caused
settings.py to load before test_mode was set, breaking electrode validation
skip logic in the CI merge commit.

Master's DockerMySQLManager.wait() doesn't touch dj.config - it just waits
for container health. Our implementation should do the same and let fixtures
(server_credentials + dj_conn) handle config setup.

Fixes electrode ID duplicate validation errors in CI tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

The connected property needs to update dj.config to ensure test_mode is set
before spyglass modules import. While dj_conn fixture also sets this, the
connected property is accessed early in mini_insert before some imports,
so it needs to redundantly update config to ensure proper initialization.

This matches master's DockerMySQLManager.connected implementation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

Root cause: When settings.py is first imported, module-level variables like
test_mode are set from sg_config at import time (line 725). Later, when test
fixtures call load_config(test_mode=True), it updates sg_config._test_mode
and dj.config['custom']['test_mode'], but the module-level variable stays stale.

The _test_mode property in BaseMixin was doing:
    from spyglass.settings import test_mode
    return test_mode  # Returns stale module-level variable!

This caused electrode validation to run during tests even though test_mode
should be True, because it was checking the stale module variable that was
set to False at initial import.

Fix: Changed _test_mode to check dj.config['custom']['test_mode'] directly
instead of importing the module-level variable. This avoids global variables
and ensures we always get the current config state.

This is why PR #1454's electrode validation worked on master but failed when
merged with our branch - subtle import order differences meant our branch
triggered settings import earlier, before fixtures could update test_mode.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

…st functions

Root cause: Module-level import of SpyglassConfig in test_config_schema.py was
causing pytest to hang during test collection. SpyglassConfig imports datajoint,
which may attempt database operations before test fixtures are set up.

Evidence:
- Master branch: "collected 497 items" appears in logs, tests complete in 56min
- Our branch: No "collected" output, hangs for 93min then fails with runner communication error
- No pytest output in logs indicates collection never completed

Fix: Moved all SpyglassConfig imports from module-level to inside individual test
functions, using lazy importing pattern. This ensures imports happen after pytest
fixtures have set up the test environment properly.

Related: #1414

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

Root cause: Tests were using fake paths like "/cli/path" and "/env/path" that
don't exist. After refactoring, get_base_directory() validates paths by creating
them and testing write permissions. Tests failed when trying to create these
fake paths.

Fix: Updated tests to use pytest's tmp_path fixture, providing real temporary
directories that can be safely created and cleaned up. Tests now verify both:
1. Path priority logic (CLI > env var)
2. Actual directory creation and validation behavior

This makes tests more realistic and tests the full behavior, not just path
resolution logic.

Related: #1414

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

The installer script is actually named install.py, not quickstart.py. Updated all references in QUICKSTART.md to use the correct filename.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

…ng database

Based on reviewer feedback (samuelbray32), reordered installer priorities to focus on the primary use case of new lab members connecting to existing databases, rather than emphasizing Docker trial setup.

Changes:
1. **Database menu reordering**: Remote database now option 1 (recommended for lab members), Docker moved to option 2 (trial/testing)
2. **Password change flow**: Added interactive password change prompt after successful remote database connection for new lab members with temporary credentials
3. **QUICKSTART.md user personas**: Added "Choose Your Path" section clearly distinguishing "Joining an Existing Lab" (primary) vs "Trying Spyglass Locally" (secondary) use cases
4. **Improved UX messaging**: Database descriptions now emphasize intended use case ("Connect to existing lab database" vs "Local trial database")

Addresses reviewer concerns:
- Primary use case: Build environment → Connect to lab DB → Change password → Store config globally ✅
- Secondary use case: Docker trial setup still supported but deprioritized
- Password security: New lab members can change temporary admin-provided credentials during setup

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

Per user feedback, password change should work during installation, not be deferred to post-install.

Implementation:
- change_database_password() now takes env_name parameter and runs DataJoint code inside conda environment via subprocess
- Uses 'conda run -n env_name python -c code' to execute in correct environment where DataJoint is installed
- Prompts user interactively with getpass for new password with confirmation
- Executes ALTER USER statement via DataJoint connection
- Returns new password which updates config before saving

Changes:
- Added env_name parameter to setup_database_remote(), handle_database_setup_interactive(), and handle_database_setup_cli()
- Updated all call sites in run_installation() to pass args.env_name through the call chain
- Removed post-install password change instructions from success message
- Updated QUICKSTART.md to reflect during-install password change flow
- Fixed terminology consistency to "lab's existing database"

This properly addresses the DataJoint import timing issue (installer runs in outer Python) while allowing password change to happen during installation as intended.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

Security Issue:
The new_password was embedded directly into the Python code string using repr()
and f-string interpolation, then used in cursor.execute(). This created a mixed
parameterization approach where the password underwent double-interpolation:
1. F-string embedding when building python_code
2. SQL execution inside the subprocess

This violated defense-in-depth principles and created unnecessary security risk.

Fix:
- Pass new_password via environment variable SPYGLASS_NEW_PASSWORD instead of
  embedding in code string
- Retrieve it inside subprocess using os.environ.get()
- Use proper SQL parameterization without any f-string interpolation:
  cursor.execute("ALTER USER %s@'%%' IDENTIFIED BY %s", (user, new_password))

This eliminates the double-interpolation issue and follows security best practices
for handling sensitive data in subprocess calls.

Credits: Identified by code-reviewer agent during security review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>